Web Survey Bibliography
Background: Traditional modes of survey data collection show decreasing response rates and increasing costs. Web surveys potentially provide a cost-effective alternative. Opt-in volunteer web panels are widely used for market research or opinion polling, but less for academic or government research because of concerns about their representativeness arising from the effect of self-selection bias. Various methods attempt to make web panel surveys more representative of the population. We compared results from four UK web surveys with Natsal-3, a national probability sample survey.
Methods: The four web surveys were done by three UK market research companies, each with large volunteer web panels. A shortened Natsal-3 questionnaire was included on four web surveys: two used basic demographic quotas and two were modified with variables correlated with key outcomes as additional quotas. Panel members aged 18–44 years, who were resident in Great Britain, and who met the criteria set for the quotas were eligible. Each company was asked to provide a sample of 2000 participants, which achieved between 2000 and 2099 participants, with data collection between May and July, 2012. After weighting for age and sex, comparisons were made with Natsal-3 for demographic characteristics, key behaviours, and opinions, to examine whether modified quotas improved the results. We calculated mean absolute odds ratios (ORs) and the percentage of variables that significantly differed from Natsal-3 at the 5% significance level. We used Z tests based on bootstrap standard errors to assess whether the modified quota surveys provided estimates closer to Natsal-3 than the basic quota surveys. We used generalised estimating equations to assess whether the two modified quota surveys and likewise the two basic quota surveys provided consistent estimates. Six demographic and 35 behaviour or opinion variables were compared, including reporting of same-sex experience and attraction, vaginal sex in the past month, number of heterosexual partners, and attendance to a sexually transmitted infection clinic.
Findings: All four web surveys gave different results from Natsal-3 for most of the variables, and overall the two surveys with modified quotas did not provide estimates that were closer to Natsal-3 than the basic quota surveys. Estimates for men from the web surveys differed more from Natsal-3 than estimates for women. For men, the mean absolute OR was 1·81 for basic and 1·64 for modified surveys, and for women the results were 1·43 for basic and 1·42 for modified surveys. The differences between the web surveys and Natsal-3 were numerically greater for questions asked face-to-face than those asked in self-completion format, but no formal statistical comparison was made. The two modified quota surveys seemed to differ from each other (ie, were inconsistent, p=0·07), as did the basic quota surveys (p<0·0001). Adjustment of web panel quota controls did not lead to much improvement according to the results of Z tests, although for men improvements were more evident than for women. Moreover, meeting the modified quotas proved difficult, and the quotas had to be relaxed in both cases.
Interpretation: When measuring sensitive sexual behaviours in the Great Britain population, volunteer web panel surveys provided significantly different estimates than a probability computer-assisted personal interview or computer-assisted self-interview survey, and modified quotas did not clearly improve estimation. At present, non-probability web panels are not an appropriate means of gathering data if accurate estimates of population prevalence are the intention, regardless of whether quotas are basic or more sophisticated. This finding is consistent with findings from similar research from the USA and other European countries.
Funding: Medical Research Council and the Wellcome Trust.
Homepage (Abstract)/ (Full text)
Further details
Web survey bibliography (290)
- Mind the Mode: Differences in Paper vs. Web-Based Survey Modes Among Women With Cancer; 2017; Hagan, T. L.; Belcher, S. M.; Donovan, H. S.
- Telephone versus Online Survey Modes for Election Studies: Comparing Canadian Public Opinion and Vote...; 2017; Breton, C.; Cutler, F.; Lachance, S.; Mierke-Zatwarnicki, A.
- Paradata as an aide to questionnaire design: Improving quality and reducing burden; 2017; Timm, E.; Stewart, J.; Sidney, I.
- Fieldwork monitoring and managing with time-related paradata; 2017; Vandenplas, C.
- Where, When, How and with What Do Panel Interviews Take Place and Is the Quality of Answers Affected...; 2017; Niebruegge, S.
- Do distractions during web survey completion affect data quality? Findings from a laboratory experiment...; 2017; Wenz, A.
- A Comparison of Emerging Pretesting Methods for Evaluating “Modern” Surveys; 2017; Geisen, E., Murphy, J.
- Pushing to web in the ISSP; 2017; Jonsdottir, G. A.; Dofradottir, A. G.; Einarsson, H. B.
- Push2web or less is more? Experimental evidence from a mixed-mode population survey at the community...; 2017; Neumann, R.; Haeder, M.; Brust, O.; Dittrich, E.; von Hermanni, H.
- Rates, Delays, and Completeness of General Practitioners’ Responses to a Postal Versus Web-Based...; 2017; Sebo, P.; Maisonneuve, H.; Cerutti, B.; Pascal Fournier, J.; Haller, D. M.
- Comparing data quality and cost from three modes of on-board transit surveys ; 2017; Agrawal, A. W.; Granger-Bevan, S.; W.; Newmark, G. L.; Nixon, H.
- Overview: Online Surveys; 2017; Vehovar, V.; Lozar Manfreda, K.
- FocusVision 2015 Annual MR Technology Report; 2016; Macer, T., Wilson, S.
- Comparisons of Online Recruitment Strategies for Convenience Samples: Craigslist, Google AdWords, Facebook...; 2016; Antoun, C., Zhang, C., Conrad, F. G., Schober, M. F.
- Feature phones no barrier to conducting an effective conjoint study ; 2016; de Rooij, R.; Dossin, R.
- Mixing Modes: Challenges (and Tradeoffs) of Adapting a Mailed Paper Survey to the Web ; 2016; Wilkinson-Flicker, S.; McPhee, C. B.; Medway, R.; Kaiser, A.; Cutts, K.
- Does Changing Monetary Incentive Schemes in Panel Studies Affect Cooperation? A Quasi-experiment on...; 2016; Schaurer, I.; Bosnjak, M.
- Survey Mode and Mail Method: A Practical Experiment in Survey Fielding for a Multi-round Survey ; 2016; Sullivan, B. D.; Duda, N.; Bogen, K.; Clusen, N. A.; Wakar, B.; Zhou, H.
- Early-bird Incentives: Results From an Experiment to Determine Response Rate and Cost Effects ; 2016; De Santis, J.; Callahan, R.; Marsh, S.; Perez-Johnson, I.
- Using Cash Incentives to Help Recruitment in a Probability Based Web Panel: The Effects on Sign Up Rates...; 2016; Krieger, U.
- Assessing Changes in Coverage Bias of Web Surveys a s Internet Access Increases in the United States...; 2016; Sterrett, D.; Malato, D.; Benz, J.; Tompson, T.; English, N.
- Thinking Inside the Box Visual Design of the Response Box Affects Creative Divergent Thinking in an...; 2016; Mohr, A. H.; Sell, A.; Lindsay, T.
- Web surveys for offline rural communities ; 2016; Gichohi, B. W.
- Adaptive survey designs to minimize survey mode effects – a case study on the Dutch Labor Force...; 2016; Calinescu, M.; Schouten, B.
- Assessment of Innovations in Data Collection Technology for Understanding Society; 2016; Couper, M. P.
- Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk; 2016; Berinsky, A.; Huber, G. A.; Lenz, G. S.
- Sample Representation and Substantive Outcomes Using Web With and Without Incentives Compared to Telephone...; 2016; Lipps, O.; Pekari, N.
- “Money Will Solve the Problem”: Testing the Effectiveness of Conditional Incentives for...; 2016; DeCamp, W.; Manierre, M. J.
- Effects of Incentive Amount and Type of Web Survey Response Rates; 2016; Coopersmith, J.; Vogel, L. K.; Bruursema, T.; Feeney, K.
- Effect of a Post-paid Incentive on Response to a Web-based Survey; 2016; Brown, J. A.; Serrato, C. A.; Hugh, M.; Kanter, M. H.; A.; Spritzer, K. L.; Hays, R. D.
- Take the money and run? Redemption of a gift card incentive in a clinician survey. ; 2016; Chen, J. S.; Sprague, B. L.; Klabunde, C. N.; Tosteson, A. N. A.; Bitton, A.; Onega, T.; MacLean, C....
- A reliability analysis of Mechanical Turk data; 2016; Rouse, S. V.
- Taming Big Data: Using App Technology to Study Organizational Behavior on Social Media; 2015; Bail, C. A.
- Using Mobile Phones for High-Frequency Data Collection; 2015; Azevedo, J. P.; Ballivian, A.; Durbin, W.
- Cell Phone and Face-to-face Interview Responses in Population-based Sur- veys - How Do They Compare?; 2015; Ghandour, L.; Ghandour, B.; Mahfoud, Z.; Mokdad, A.; Sibai, A. M.
- The Cathie Marsh lecture: What does the failure of the polls tell us about the future of survey research...; 2015; Sturgis, P., Matheson, J.
- Mixed mode surveys ; 2015; Burton, J.
- Understanding Society Innovation Panel Wave 7: Results from Methodological Experiments; 2015; Blom, A. G.; Burton, J.; Booker, C. L.; Cernat, A.; Fairbrother, M.; Jaeckle, A.; Kaminska, O.; Keusch...
- Response Effects of Prenotification, Prepaid Cash, Prepaid Vouchers, and Postpaid Vouchers: An Experimental...; 2015; van Veen, F.; Goeritz, A.; Sattler, S.
- Tailored fieldwork design to increase representative household survey response: an experiment in the...; 2015; Luiten, A.; Schouten, B.
- Using Internet to Recruit Immigrants with Language and Culture Barriers for Tobacco and Alcohol Use...; 2015; Carlini, B. H.; Safioti, L.; Rue, T. C.; Miles, L.
- Online Recruitment Methods for Web-Based and Mobile Health Studies: A Review of the Literature; 2015; Lane, T. S.; Armin, J.; Gordon, Ju. S.
- iTunes Song-Gifting is a Low-Cost, Efficient Recruitment Tool to Engage High-Risk MSM in Internet Research...; 2015; Holland, C. M.; Ritchie, N. D.; Du Bois, S. N.
- A Note on the Use of Survey Research Firms to Enable Empirical Data Collection; 2015; Schoenherr, T.; Ellram, L. M.; Tate, W. L.
- Recruiting Online: Lessons From a Longitudinal Survey of Contraception and Pregnancy Intentions of Young...; 2015; Harris, M. L.; Loxton, D.; Wigginton, B.; Lucke, J. C.
- Recruiting for addiction research via Facebook; 2015; Thornton, L. K.; Harris, K.; Baker, A.; Johnson, M.; Kay-Lambkin, F. J.
- The effectiveness of incentives on recruitment and retention rates: an experiment in a web survey; 2015; Mulder, J.; Douhou, S.
- Adding a Web Mode to Phone Surveys: Effectiveness and Cost Implications; 2015; Beebe, T. J.; Lien, R.; Luxenberg, H.; Rainey, J.
- Query on Data Collection for Social Surveys; 2014; Blanke, K., Luiten, A.
- Examining the Effect of Prenotification Postcards on Online Survey Response Rate in a University Graduate...; 2014; Lalasz, C. B., Doane, M. J., Springer, V. A., Dahir, V. B.